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Abstract 

In  order  to  meet  the  requirements  of  ISO  17025  and  the  demand  of  TAF  (Taiwan 
Accreditation  Foundation )  for  calibration  inter-laboratory  comparisons,  the  National  Time  and 
Frequency  Standard  Laboratory  (TL)  [1],  playing  the  role  of  coordinating  lab,  has  periodically 
organized  the  proficiency  testing  activities  to  provide  opportunities  for  capability  comparisons  of 
domestic  frequency  calibration  laboratories.  The  latest  two  activities  were  performed  in  2006  [2] 
and  this  year  (2009).  The  “Device  under  Test”  (DUT),  together  with  TL’s  calibration  system, 
was  transferred  to  each  participating  laboratory  according  to  a  predetermined  schedule.  The 
DUT  was  measured  by  the  participating  lab ’s  and  TL ’s  calibration  systems  simultaneously.  The 
measured  results  from  both  systems  were  then  analyzed  and  compared. 

There  were  12  and  15  participating  labs  joined  the  activities  performed  in  2006  and  2009, 
respectively.  All  of  them  were  TAF -accredited  laboratories.  As  usual,  the  En  value  was  used  to 
evaluate  a  lab’s  capability  of  calibrating  equipment  within  its  accredited  measurement 
uncertainty. 

In  these  two  activities,  the  absolute  En  values  of  all  labs  were  smaller  than  “1,”  which 
means  their  calibration  ability  was  completely  qualified.  In  this  paper,  the  related  details  of 
these  two  activity  and  the  result  comparisons  between  activities  in  2006  and  2009  are  illustrated. 


INTRODUCTION 

The  goal  of  performing  the  proficiency  testing  activity  is  to  provide  a  chance  for  the  domestic  accredited 
laboratories  to  compare  their  calibration  capability  with  one  another.  When  performing  on-site  evaluation, 
an  assessment  team  is  organized  to  examine  the  technical  competence  of  the  labs  and  their  compliance 
with  the  requirements  of  ISO/IEC  17025  General  requirements  for  the  competence  of  testing  and 
calibration  laboratories.  The  proficiency  testing  results  are  then  important  information  for  the  technical 
assessment  team  to  evaluate  the  ability  of  the  laboratory. 

In  proficiency  testing  activities  [3,4],  the  ability  of  each  individual  laboratory  is  supposed  to  be 
determined  and  able  to  achieve  its  accredited  level  of  measurement  uncertainty.  In  the  case  that  the 
performance  of  the  “Device  under  Test”  (DUT)  has  good  repeatability  and  stability,  its  reference  value 
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will  be  assigned  by  the  coordinating  lab  and  then  compared  with  measurement  results  of  the  participating 
labs,  as  shown  in  Fig  1.  When  the  reference  value  is  located  within  a  lab’s  uncertainty  range  (e.g.,  lab  1, 
2,  and  3  in  Fig.  1),  we  say  that  this  lab  works  within  its  capability  and  its  accredited  level  of  measurement 
uncertainty  is  suitable  for  the  DUT  being  calibrated. 


Ref 

Value 


Uncertainty  Range 


^5)  =Results  of  LAB  N 


Fig.  1 .  Measurement  results  and  uncertainty  ranges  of  participating  labs. 


However,  the  characteristic  of  the  DUT  in  the  time/frequency  area  (such  as  a  frequency  oscillator)  is  non¬ 
stationary,  so  the  reference  value  may  change  gradually.  Therefore,  together  with  the  DUT,  a  calibration 
system  of  the  coordinating  lab  is  usually  transferred  to  each  participant  laboratory  to  get  the  relative 
reference  value  simultaneously  with  the  measurement.  In  other  words,  the  DUT  is  measured  by  the 
participant’s  and  the  coordinating  lab’s  calibration  systems  at  the  same  time. 


EVALUATION  OF  RESULTS 


In  the  proficiency  test  activities  among  calibration  laboratories,  the  En  value  is  adopted  to  indicate  how 
well  labs  are  within  their  particular  measurement  uncertainty,  taking  account  of  the  measurement 
uncertainty  of  the  reference  value.  En  stands  for  Error  normalized  and  is  defined  as: 


E  = 


Z-Z 


ref 


C/952(Z,)+C/952(Z  .) 


(1) 


where  Z,  is  die  participant  laboratory ’s  result 

Xref  is  the  coordinating  laboratory ’s  result 

U 9j(Z,)  is  the  participating  laboratory’s  reported  uncertainty  (95%) 
Ug5{Xref)  is  the  coordinating  laboratory’s  reported  uncertainty  (95%) 

|En  |  <  1  indicates  that  the  result  and  the  reference  value  are  in  agreement 

|E n  |  >  1  indicates  that  the  result  is  different  from  the  reference  value. 


Note  that  the  calculation  of  En  value  does  not  intend  to  indicate  which  lab’s  result  is  closest  to  the 
reference  value,  as  high-level  calibration  labs  may  have  their  En  values  similar  to  that  of  some  other  labs 
with  both  larger  measurement  errors  and  uncertainties. 

Besides,  one  should  be  aware  that  the  reference  value  itself  has  a  measurement  uncertainty.  The 
coordinating  lab  should  have  the  capability  to  give  a  better  measurement  uncertainty  than  the  participating 
lab’s;  otherwise,  it  will  be  difficult  to  evaluate  each  lab’s  performance.  Consequently,  the  |En|  <  1  limit 
really  only  represents  the  cutoff,  below  which  it  is  likely  that  the  result  is  acceptable  and  above  which  it  is 
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unlikely  that  the  result  is  acceptable.  So  when  considering  any  result  with  En  greater  than  1,  all  factors 

should  be  evaluated  to  see  if  there  is  a  systematic  bias  that  is  consistently  positive  or  consistently  negative 
which  causes  the  problem. 


PROFICIENCY  TESTING  ACTIVITY 

TL,  playing  the  role  of  coordinating  lab,  has  periodically  organized  the  proficiency  testing  activities  every 
3  years  since  2003.  The  latest  two  activities  were  performed  in  2006  and  2009,  with  HP8662A  [5]  and 
HP33250A  frequency  synthesizers  as  the  DUT,  respectively.  Both  HP8662A  and  HP33250A  can  offer 
5/10  MHz  output  signals,  so  any  laboratory  with  a  calibration  capability  for  either  5  MHz  or  10  MHz  can 
join  these  activities.  The  aging  rate  of  HP8662A  is  smaller  than  5.0x10  10/day,  which  is  lower  than  that  of 
HP33250A,  which  is  about  5.5x10  9/day. 

Since  the  properties  of  an  oscillator’s  signal  are  non-stationary,  TL’s  calibration  system,  consisting  of  a 
5071 A  cesium  clock,  a  SR620  time -interval  counter,  and  a  recording  computer  along  with  the  DUT,  were 
sent  to  each  participating  lab  sequentially.  The  DUT’s  reference  value  can  then  be  obtained  during  the 
measurement  process  in  each  lab.  The  above  mentioned  5071 A  cesium  clock  is  TL’s  portable  frequency 
reference,  which  is  traceable  to  the  national  frequency  standard.  To  make  sure  that  the  HP5071A  can 
satisfy  the  requirement  of  traceability,  the  performance  of  this  Cs  clock  was  measured  in  TL  before  and 
after  every  measurement  trip  to  the  participating  labs.  The  block  diagram  of  the  related  system  for  the 
proficiency  testing  activities  is  shown  in  Fig.  2. 


Fig.  2.  The  system  arrangement  in  proficiency  testing  activities. 


The  number  of  participating  labs  that  joined  the  proficiency  testing  activities  was  12  in  2006  and 
increased  to  15  in  2009,  but  the  same  procedure  was  followed.  Before  the  measurement  Lips  to  the 
participating  labs  in  each  activity,  an  opening  meeting  was  held  in  advance  to  discuss  all  details  about  the 
related  activities.  A  consensus  referring  to  the  schedule  for  transferring  the  DUT  and  TL’s  calibration 
system,  the  measurement  method,  the  uncertainty  evaluation,  the  result  expression,  etc.  was  reached. 
Since  2006,  TL  has  recommended  that  the  participating  labs  include  the  best  measurement  capability  of 
the  individual  lab,  the  DUT’s  aging  rate  and  temperature  effects,  etc.  in  their  uncertainty  budgets. 
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Note  that,  in  general  calibration  services,  an  uncertainty  budget  usually  included  the  DUT’s  aging  to 
assure  the  validity  of  the  measurement  result  during  the  period  that  the  DUT  returned  to  its  lab  till  the 
next  time  the  DUT  was  sent  for  calibration.  However,  it  isn’t  necessary  to  count  the  aging  for  a  long 
period  (for  example,  1  year)  in  proficiency  testing  activities,  because  the  DUT  only  serves  as  an 
intermediate  for  proficiency  testing  comparisons  and  the  measurement  time  in  each  participating  labs  is 
not  longer  than  1  hour.  Therefore,  we  announced  DUT’s  aging  rate  for  1  day  to  each  participating  lab  for 
their  evaluation  of  measurement  uncertainty.  While  finishing  the  measurement,  each  participating  lab 
was  required  to  send  its  test  report,  raw  measurement  data,  and  uncertainty  budget  to  TL  within  3  days. 

After  the  measurement  trips  to  the  participating  labs,  collecting  the  measurement  data  and  evaluating  the 
capability  of  each  laboratory,  a  meeting  was  held  for  final  discussion  and  providing  the  chance  for  face- 
to-face  communication  with  the  participant  labs. 


RESULTS  ANALYSIS  AND  CONCLUSION 


The  proficiency  testing  results  of  12  participating  labs  in  2006  and  15  participating  labs  in  this  year  are 
shown  in  Tables  1  and  2,  respectively.  X;,  Xref,  Uy.TX.j  and  U95(Xref)  stand  for  the  corresponding 
measurement  results  (frequency  accuracy)  and  reported  uncertainties  from  participating  labs  and  TL. 
Using  these  four  parameters,  the  En  value  of  each  participating  lab  could  be  obtained. 

Considering  a  drifting  frequency  standard  may  influence  the  measurement  result,  it  is  reasonable  for 
participant  labs  to  include  the  latest  calibrated  accuracy  of  their  individual  frequency  standards  in  the  X, 
measurements,  or  separately  specify  both  of  them  in  each  participant  lab’s  test  report.  Results  of  the  latter 
are  adopted  in  Table  1  and  the  mentioned  calibrated  accuracy  AX,  can  be  used  for  the  En  correction. 

The  corrected  En  value  could  be  obtained  using  equation  as  below: 


E 


n 


(X,.+AX,.)-Xrg/ 

^/M2(X,)  +  t/M2(X^)' 


(2) 


Besides,  adjustment  of  Xref  is  negligible  because  the  calibrated  accuracy  of  TL’s  portable  frequency 
reference  can  reach  1.5x10  13,  about  4  orders  lower  than  the  magnitude  of  Xref  itself. 

For  most  of  the  participants  in  Table  1,  the  corrected  |En|  are  smaller  than  or  equal  to  the  original  ones. 

This  shows  that  the  En  correction  is  meaningful,  which  gives  more  reasonable  results  for  inter-laboratory 
comparisons.  Equation  (2)  can  also  be  considered  as  the  corrected  measurement  difference  between  a 
participating  lab  and  TL,  divided  by  their  combined  uncertainty. 

In  general,  Uy.TXjj  is  the  dominant  term  in  the  combined  uncertainty,  so  the  corrected  |En|  is  a  good 
index  to  show  how  well  a  participating  lab  is  within  its  accredited  capability.  All  the  corrected  |En  |  of  12 

participating  labs  in  2006  and  15  participating  labs  this  year  are  smaller  than  “1,”  which  indicates  that  the 
calibration  capabilities  of  all  the  participants  are  qualified  in  these  two  activities. 

Moreover,  one  may  found  that  most  of  the  corrected  lEnl  values  in  Table  2  are  smaller  than  those  in  Table 
1.  This  is  caused  by  the  performances  of  different  DUTs  adopted  in  these  two  activities.  The  aging  rate 
and  temperature  coefficient  of  an  HP33520A  (DUT  of  2009)  are  larger  than  those  of  an  HP8662A  (DUT 
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of  2006),  so  the  combined  uncertainties  are  commonly  larger  and  the  correspondent  lEnl  values  become 
smaller.  Generally  speaking,  a  DUT  with  smaller  aging  and  temperature  coefficient  would  be  more 
suitable  for  the  proficiency  testing  application,  and  the  capability  of  the  participating  labs  can  be  more 
properly  evaluated. 
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Table  1.  Proficiency  testing  results  of  12  participating  labs. 


Report  No. 

Xj 

xref 

AX; 

U95(Xi) 

U95(Xref) 

IEJ 

lEJ(corrected) 

TL-PT2-01 

-3.0343E-09 

-4.30E-09 

-2.6E-10 

1.19E-09 

3.76E-10 

1.01 

0.81 

TL-PT2-02 

-3.8000E-09 

-3.70E-09 

6.4E-12 

4.10E-10 

4.68E-10 

0.16 

0.15 

TL-PT2-03 

2.5507E-09 

3.00E-09 

3.2E-10 

7.32E-10 

3.76E-10 

0.55 

0.16 

TL-PT2-04 

-1.0000E-08 

-4.30E-09 

3.3E-11 

5.82E-09 

6.83E-10 

0.97 

0.97 

TL-PT2-05 

-3.0800E-09 

-2.90E-09 

-1.9E-13 

6.18E-09 

3.08E-10 

0.03 

0.03 

TL-PT2-06 

-4.5250E-09 

-5.50E-09 

-1.0E-09 

3.16E-09 

3.76E-10 

0.31 

0.01 

TL-PT2-07 

-4.4500E-08 

-1.40E-09 

9.3E-09 

8.96E-08 

4.68E-10 

0.48 

0.38 

TL-PT2-08 

-4.6000E-09 

-4.90E-09 

1.3E-10 

9.00E-10 

3.76E-10 

0.31 

0.44 

TL-PT2-09 

-2.3400E-09 

-1.60E-09 

1.4E-10 

9.13E-10 

3.38E-10 

0.76 

0.62 

TL-PT2-10 

-3.6330E-09 

-3.30E-09 

1.5E-12 

5.05E-10 

3.76E-10 

0.53 

0.53 

TL-PT2-1 1 

-5.6900E-09 

-4.74E-09 

-7.6E-1 1 

9.20E-08 

3.76E-10 

0.01 

0.01 

TL-PT2-12 

-3.0000E-09 

-2.10E-09 

-3.2E-10 

3.30E-07 

3.76E-10 

0.01 

0.01 
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Table  2.  Proficiency  testing  results  of  15  participating  labs. 


Report  No. 

Xj 

xref 

AX; 

U95(X.) 

U95(Xref) 

1  En  l(corrected) 

TL-98FMPT-01 

-4.918564E-07 

-4.92345E-07 

-1.70E-13 

7.571E-08 

6.93E-08 

0.005 

TL-98FMPT-02 

-3.177987E-07 

-3.17600E-07 

-1.40E-1 1 

1.500E-07 

6.93E-08 

0.001 

TL-98FMPT-03 

-3.841748E-07 

-4.23000E-07 

3.20E-08 

1.500E-06 

6.93E-08 

0.005 

TL-98FMPT-04 

-4.1 15972E-07 

-4.1 1000E-07 

-2.40E-10 

4.900E-07 

6.93E-08 

0.001 

TL-98FMPT-05 

-5.097840E-07 

-5.10000E-07 

1.45E-10 

4.200E-07 

6.93E-08 

0.001 

TL-98FMPT-06 

-3.260692E-07 

-3.20000E-07 

3.70E-10 

1.400E-07 

6.93E-08 

0.041 

TL-98FMPT-07 

-4.194414E-07 

-4.21000E-07 

8.10E-11 

3.300E-07 

6.93E-08 

0.004 

TL-98FMPT-08 

1.112796E-08 

1.11000E-08 

-1.50E-13 

4.300E-08 

6.93E-08 

0.001 

TL-98FMPT-09 

-5.255941E-07 

-5.25000E-07 

-5.00E-10 

1.400E-07 

6.93E-08 

0.001 

TL-98FMPT-10 

-4.96463  IE-07 

-4.96185E-07 

1.50E-10 

1.200E-06 

6.93E-08 

0.001 

TL-98FMPT-1 1 

-3.686349E-07 

-3.68000E-07 

-1.20E-09 

1.390E-07 

6.93E-08 

0.004 

TL-98FMPT-12 

-3.432065E-07 

-3.42923E-07 

-1.00E-12 

1.390E-07 

6.93E-08 

0.002 

TL-98FMPT-13 

-3.316300E-07 

-3.37000E-07 

6.50E-09 

2.200E-07 

6.93E-08 

0.005 

TL-98FMPT-14 

-3.299086E-07 

-3.30000E-07 

9.90E-11 

6.820E-08 

6.93E-08 

0.001 

TL-98FMPT-15 

-3.656232E-07 

-3.94400E-07 

6.00E-10 

6.820E-08 

6.93E-08 

0.29 
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